Skip to content

[feature](index change)Support light index change for inverted index without parser#52251

Merged
airborne12 merged 14 commits intoapache:masterfrom
airborne12:add-cloud-index
Jun 30, 2025
Merged

[feature](index change)Support light index change for inverted index without parser#52251
airborne12 merged 14 commits intoapache:masterfrom
airborne12:add-cloud-index

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Jun 24, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This PR adds support for lightweight index add for inverted indexes without parser in cloud mode and extends the existing “light” mode for NGRAM Bloom Filter indexes. And also introduces a new session variable (enable_add_index_for_new_data) to control the above logic.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12
Copy link
Member Author

airborne12 commented Jun 24, 2025

run buildall

@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

1 similar comment
@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 34178 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 518b8a4234fbf5a20f6ea7f4ac3036c461738fce, data reload: false

------ Round 1 ----------------------------------
q1	15043	5204	5063	5063
q2	1599	285	196	196
q3	8523	1314	730	730
q4	9860	1015	526	526
q5	7723	2382	2384	2382
q6	183	173	138	138
q7	903	764	606	606
q8	9311	1298	1076	1076
q9	6798	5096	5063	5063
q10	6889	2386	1955	1955
q11	484	296	293	293
q12	348	343	219	219
q13	17786	3701	3081	3081
q14	229	225	226	225
q15	537	475	482	475
q16	437	430	370	370
q17	621	885	388	388
q18	7606	7324	7190	7190
q19	1267	955	536	536
q20	363	341	228	228
q21	4097	3246	2446	2446
q22	1049	1043	992	992
Total cold run time: 101656 ms
Total hot run time: 34178 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5168	5060	5106	5060
q2	277	328	227	227
q3	2153	2684	2334	2334
q4	1370	1778	1343	1343
q5	4292	4153	4129	4129
q6	207	168	129	129
q7	1893	1854	1692	1692
q8	2516	2383	2465	2383
q9	6859	6840	6800	6800
q10	2966	3175	2703	2703
q11	561	504	477	477
q12	649	741	581	581
q13	3326	3699	3052	3052
q14	269	299	265	265
q15	517	467	473	467
q16	439	477	432	432
q17	1131	1449	1346	1346
q18	7322	7226	7033	7033
q19	793	812	905	812
q20	1916	1995	1805	1805
q21	4814	4248	4299	4248
q22	1055	1031	1033	1031
Total cold run time: 50493 ms
Total hot run time: 48349 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 185388 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 518b8a4234fbf5a20f6ea7f4ac3036c461738fce, data reload: false

query1	996	389	392	389
query2	6545	1841	1799	1799
query3	6740	222	217	217
query4	26082	23175	22862	22862
query5	4341	631	490	490
query6	313	217	210	210
query7	4636	499	293	293
query8	290	234	215	215
query9	8640	2620	2629	2620
query10	487	336	274	274
query11	15264	15142	14833	14833
query12	175	123	106	106
query13	1652	543	437	437
query14	10078	6050	6149	6050
query15	201	185	182	182
query16	7645	625	439	439
query17	1585	764	549	549
query18	2019	397	291	291
query19	202	204	155	155
query20	115	122	114	114
query21	212	123	111	111
query22	4073	4119	3957	3957
query23	33845	33077	33276	33077
query24	8407	2354	2402	2354
query25	522	448	381	381
query26	1230	269	147	147
query27	2735	513	338	338
query28	4318	2122	2107	2107
query29	732	536	426	426
query30	287	211	196	196
query31	933	821	751	751
query32	71	69	64	64
query33	552	384	311	311
query34	791	845	539	539
query35	775	827	728	728
query36	958	968	904	904
query37	122	96	81	81
query38	4083	4119	4085	4085
query39	1511	1480	1452	1452
query40	210	121	105	105
query41	64	65	63	63
query42	131	119	112	112
query43	502	533	485	485
query44	1301	834	833	833
query45	184	191	172	172
query46	834	1026	639	639
query47	1735	1759	1717	1717
query48	396	429	309	309
query49	751	496	412	412
query50	660	666	396	396
query51	4106	4165	4154	4154
query52	111	110	102	102
query53	233	251	186	186
query54	575	563	498	498
query55	82	90	85	85
query56	307	307	277	277
query57	1217	1181	1119	1119
query58	266	267	257	257
query59	2511	2680	2567	2567
query60	329	325	311	311
query61	125	122	136	122
query62	793	708	673	673
query63	221	183	186	183
query64	4223	984	649	649
query65	4257	4158	4156	4156
query66	1079	427	353	353
query67	15843	15697	15511	15511
query68	6148	910	527	527
query69	476	304	270	270
query70	1210	1072	1079	1072
query71	422	316	309	309
query72	5449	4804	4868	4804
query73	710	690	351	351
query74	9072	9140	8795	8795
query75	3169	3223	2707	2707
query76	3255	1200	743	743
query77	474	381	289	289
query78	9939	10078	9433	9433
query79	1214	814	584	584
query80	644	516	430	430
query81	480	254	227	227
query82	186	129	96	96
query83	257	258	236	236
query84	256	104	96	96
query85	756	396	307	307
query86	320	310	310	310
query87	4384	4615	4370	4370
query88	2848	2281	2253	2253
query89	367	317	282	282
query90	1734	212	205	205
query91	139	140	114	114
query92	62	60	56	56
query93	1044	944	587	587
query94	609	388	303	303
query95	371	288	287	287
query96	489	569	275	275
query97	2695	2773	2641	2641
query98	236	210	201	201
query99	1337	1388	1248	1248
Total cold run time: 267961 ms
Total hot run time: 185388 ms

@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 33843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85faf531f18ee5bf176684b3145e5d6c6d04b0a3, data reload: false

------ Round 1 ----------------------------------
q1	17616	5188	5036	5036
q2	1949	287	188	188
q3	10376	1248	692	692
q4	10239	989	527	527
q5	7548	2306	2364	2306
q6	177	163	133	133
q7	902	742	598	598
q8	9318	1262	1181	1181
q9	6726	5047	5129	5047
q10	6936	2400	1960	1960
q11	494	290	284	284
q12	346	351	217	217
q13	17775	3666	3080	3080
q14	229	223	216	216
q15	555	470	486	470
q16	437	433	369	369
q17	618	879	376	376
q18	7466	7090	7164	7090
q19	2151	1025	542	542
q20	330	343	216	216
q21	3667	3164	2361	2361
q22	1054	1035	954	954
Total cold run time: 106909 ms
Total hot run time: 33843 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5249	5126	5092	5092
q2	254	317	222	222
q3	2183	2633	2320	2320
q4	1354	1791	1321	1321
q5	4281	4353	4378	4353
q6	207	169	127	127
q7	2015	1935	1818	1818
q8	2613	2563	2514	2514
q9	7083	7091	7061	7061
q10	2995	3253	2843	2843
q11	561	506	498	498
q12	692	760	584	584
q13	3555	3866	3248	3248
q14	289	319	284	284
q15	534	466	467	466
q16	449	497	409	409
q17	1098	1420	1359	1359
q18	7389	7121	7084	7084
q19	775	782	809	782
q20	1872	1964	1780	1780
q21	4788	4414	4291	4291
q22	1067	1061	1009	1009
Total cold run time: 51303 ms
Total hot run time: 49465 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 185998 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85faf531f18ee5bf176684b3145e5d6c6d04b0a3, data reload: false

query1	973	411	403	403
query2	6556	1797	1820	1797
query3	6741	217	214	214
query4	26179	23631	23346	23346
query5	4806	651	486	486
query6	303	227	207	207
query7	4647	517	293	293
query8	277	237	213	213
query9	8679	2623	2648	2623
query10	514	335	276	276
query11	15242	15396	14737	14737
query12	182	113	114	113
query13	1672	534	400	400
query14	10065	6131	6098	6098
query15	203	204	178	178
query16	7652	619	498	498
query17	1259	742	602	602
query18	2027	407	319	319
query19	209	195	168	168
query20	123	117	117	117
query21	219	128	121	121
query22	4260	4391	4181	4181
query23	33980	32993	33340	32993
query24	8363	2350	2378	2350
query25	518	453	385	385
query26	805	270	151	151
query27	2685	513	336	336
query28	4279	2122	2121	2121
query29	677	546	422	422
query30	279	216	198	198
query31	918	859	770	770
query32	70	68	63	63
query33	549	370	301	301
query34	827	862	524	524
query35	787	815	761	761
query36	934	993	894	894
query37	113	99	77	77
query38	4141	4187	4093	4093
query39	1478	1427	1423	1423
query40	210	121	109	109
query41	63	57	62	57
query42	125	107	113	107
query43	479	517	481	481
query44	1348	825	821	821
query45	179	179	168	168
query46	841	1023	630	630
query47	1769	1803	1769	1769
query48	382	422	332	332
query49	708	499	379	379
query50	650	661	405	405
query51	4182	4120	4100	4100
query52	114	107	93	93
query53	231	254	182	182
query54	576	560	502	502
query55	88	79	82	79
query56	321	318	278	278
query57	1170	1204	1129	1129
query58	268	275	259	259
query59	2660	2680	2564	2564
query60	329	324	310	310
query61	123	123	123	123
query62	841	749	661	661
query63	230	190	179	179
query64	3380	994	673	673
query65	4222	4204	4179	4179
query66	906	413	311	311
query67	15750	15728	15382	15382
query68	9276	943	527	527
query69	485	320	269	269
query70	1211	1103	1108	1103
query71	470	318	312	312
query72	5214	4702	4568	4568
query73	670	553	349	349
query74	9182	9224	8914	8914
query75	4336	3189	2709	2709
query76	3591	1188	746	746
query77	990	390	298	298
query78	9992	10322	9418	9418
query79	1106	822	576	576
query80	742	520	450	450
query81	487	266	218	218
query82	416	124	99	99
query83	291	250	241	241
query84	289	106	96	96
query85	787	350	397	350
query86	336	292	283	283
query87	4437	4422	4386	4386
query88	2876	2353	2281	2281
query89	388	322	283	283
query90	1921	211	206	206
query91	141	159	115	115
query92	75	62	59	59
query93	1047	932	583	583
query94	676	394	287	287
query95	371	291	278	278
query96	499	572	277	277
query97	2714	2772	2675	2675
query98	237	205	199	199
query99	1442	1391	1329	1329
Total cold run time: 273031 ms
Total hot run time: 185998 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85faf531f18ee5bf176684b3145e5d6c6d04b0a3, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.25	0.06	0.07
query4	1.61	0.10	0.10
query5	0.42	0.42	0.42
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.50	0.53
query10	0.57	0.58	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.63	0.61	0.61
query14	0.81	0.82	0.85
query15	0.89	0.89	0.90
query16	0.39	0.39	0.41
query17	1.09	1.12	1.04
query18	0.23	0.22	0.22
query19	1.95	1.82	1.88
query20	0.02	0.02	0.02
query21	15.37	0.91	0.54
query22	0.77	1.20	0.75
query23	14.78	1.37	0.63
query24	7.07	1.58	1.47
query25	0.47	0.18	0.08
query26	0.64	0.15	0.15
query27	0.06	0.05	0.05
query28	9.94	0.88	0.46
query29	12.54	4.02	3.27
query30	0.25	0.10	0.07
query31	2.82	0.60	0.38
query32	3.23	0.55	0.46
query33	3.05	3.08	3.06
query34	16.14	5.38	4.76
query35	4.83	4.76	4.80
query36	0.71	0.53	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.21 s
Total hot run time: 30.19 s

@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

@airborne12 airborne12 requested a review from Copilot June 25, 2025 10:48

This comment was marked as outdated.

Sign in to view

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 34254 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49c4f42c8728f9de8061edf7eb76395f6963f82a, data reload: false

------ Round 1 ----------------------------------
q1	17699	5256	5119	5119
q2	1950	286	185	185
q3	10528	1329	741	741
q4	10254	998	544	544
q5	7846	2367	2362	2362
q6	187	164	137	137
q7	922	765	605	605
q8	9320	1276	1134	1134
q9	6747	5038	5116	5038
q10	6898	2374	1978	1978
q11	497	296	294	294
q12	339	352	221	221
q13	17768	3676	3106	3106
q14	238	234	216	216
q15	551	487	476	476
q16	431	428	380	380
q17	589	882	362	362
q18	7606	7315	7219	7219
q19	1531	963	549	549
q20	321	339	231	231
q21	3663	3194	2365	2365
q22	1031	1021	992	992
Total cold run time: 106916 ms
Total hot run time: 34254 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5191	5133	5149	5133
q2	248	324	220	220
q3	2195	2678	2311	2311
q4	1374	1797	1335	1335
q5	4286	4467	4392	4392
q6	220	176	137	137
q7	2043	1948	1775	1775
q8	2615	2569	2541	2541
q9	7161	7165	7087	7087
q10	3046	3257	2864	2864
q11	577	546	500	500
q12	670	798	646	646
q13	3510	3809	3299	3299
q14	299	300	267	267
q15	515	472	485	472
q16	468	477	454	454
q17	1169	1427	1382	1382
q18	7502	7229	7120	7120
q19	790	820	990	820
q20	1916	2025	1913	1913
q21	4787	4385	4306	4306
q22	1085	1041	968	968
Total cold run time: 51667 ms
Total hot run time: 49942 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 186571 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49c4f42c8728f9de8061edf7eb76395f6963f82a, data reload: false

query1	962	393	402	393
query2	6550	1875	1834	1834
query3	6747	218	214	214
query4	26271	23350	23075	23075
query5	4328	602	479	479
query6	295	200	186	186
query7	4620	494	282	282
query8	259	237	212	212
query9	8594	2629	2635	2629
query10	474	341	264	264
query11	15645	15227	14841	14841
query12	167	126	107	107
query13	1649	521	411	411
query14	9693	6235	6200	6200
query15	204	203	184	184
query16	7654	635	476	476
query17	1198	721	588	588
query18	2024	433	307	307
query19	209	209	161	161
query20	124	116	117	116
query21	217	138	112	112
query22	4003	4254	4046	4046
query23	34035	33087	33067	33067
query24	8487	2420	2439	2420
query25	558	501	429	429
query26	1235	272	156	156
query27	2750	516	353	353
query28	4285	2151	2142	2142
query29	766	582	471	471
query30	292	224	210	210
query31	935	857	817	817
query32	76	67	69	67
query33	560	363	308	308
query34	791	863	543	543
query35	767	824	744	744
query36	958	976	882	882
query37	115	100	75	75
query38	4192	4159	4228	4159
query39	1472	1466	1459	1459
query40	216	116	107	107
query41	64	62	59	59
query42	132	112	111	111
query43	477	509	473	473
query44	1344	820	824	820
query45	178	182	165	165
query46	833	1031	630	630
query47	1733	1803	1696	1696
query48	378	422	317	317
query49	757	489	418	418
query50	637	685	410	410
query51	4169	4132	4159	4132
query52	110	109	105	105
query53	235	259	181	181
query54	575	583	511	511
query55	93	84	85	84
query56	312	337	285	285
query57	1147	1196	1087	1087
query58	261	272	267	267
query59	2564	2705	2609	2609
query60	323	306	320	306
query61	129	138	138	138
query62	805	709	660	660
query63	223	189	193	189
query64	4256	1009	723	723
query65	4338	4251	4173	4173
query66	1058	420	316	316
query67	15816	15732	15528	15528
query68	8803	882	539	539
query69	464	307	278	278
query70	1166	1146	1129	1129
query71	478	316	302	302
query72	5263	4760	4619	4619
query73	717	600	354	354
query74	8918	9130	9049	9049
query75	4219	3217	2705	2705
query76	3720	1199	773	773
query77	793	450	295	295
query78	10022	10211	9294	9294
query79	1827	864	595	595
query80	608	518	441	441
query81	478	262	225	225
query82	437	128	102	102
query83	278	253	236	236
query84	276	105	90	90
query85	794	363	327	327
query86	335	300	298	298
query87	4346	4454	4387	4387
query88	3492	2276	2266	2266
query89	377	312	294	294
query90	1964	214	222	214
query91	139	147	117	117
query92	72	65	62	62
query93	1325	947	587	587
query94	675	419	323	323
query95	376	293	285	285
query96	488	562	282	282
query97	2702	2780	2703	2703
query98	233	225	206	206
query99	1412	1385	1275	1275
Total cold run time: 274521 ms
Total hot run time: 186571 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 29.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49c4f42c8728f9de8061edf7eb76395f6963f82a, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.44	0.42	0.41
query6	1.17	0.66	0.66
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.51
query10	0.57	0.57	0.58
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.60	0.62
query14	0.79	0.82	0.79
query15	0.88	0.84	0.85
query16	0.38	0.38	0.41
query17	1.04	1.03	1.02
query18	0.22	0.22	0.21
query19	1.97	1.85	1.92
query20	0.01	0.01	0.01
query21	15.40	0.90	0.53
query22	0.77	1.21	0.72
query23	14.83	1.38	0.62
query24	7.15	0.96	0.38
query25	0.47	0.20	0.15
query26	0.60	0.17	0.12
query27	0.05	0.06	0.05
query28	9.36	0.95	0.46
query29	12.65	4.06	3.34
query30	0.25	0.10	0.07
query31	2.82	0.59	0.39
query32	3.23	0.55	0.46
query33	3.18	3.08	3.08
query34	16.03	5.37	4.74
query35	4.80	4.81	4.80
query36	0.67	0.49	0.48
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.76 s
Total hot run time: 29.06 s

@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

1 similar comment
@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 33949 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ce69eb4283ab724b1e5b726c5a5c5e395b364f3, data reload: false

------ Round 1 ----------------------------------
q1	17572	5185	5037	5037
q2	1951	299	189	189
q3	10287	3120	738	738
q4	10227	1035	514	514
q5	7547	2421	2328	2328
q6	182	162	136	136
q7	915	736	599	599
q8	9319	1508	1078	1078
q9	7000	5156	5090	5090
q10	6916	2397	1963	1963
q11	484	283	289	283
q12	348	353	218	218
q13	17775	3672	3126	3126
q14	241	236	220	220
q15	544	488	476	476
q16	430	424	372	372
q17	578	879	353	353
q18	8002	7264	7111	7111
q19	1227	941	557	557
q20	342	328	222	222
q21	3694	3302	2377	2377
q22	1031	1009	962	962
Total cold run time: 106612 ms
Total hot run time: 33949 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5420	5102	5083	5083
q2	252	324	223	223
q3	2147	2670	2285	2285
q4	1335	1814	1348	1348
q5	4395	4560	4415	4415
q6	254	174	127	127
q7	1946	1898	1742	1742
q8	2564	2526	2477	2477
q9	7227	7081	7207	7081
q10	3098	3255	2797	2797
q11	598	523	496	496
q12	673	745	614	614
q13	3503	3967	3206	3206
q14	272	306	283	283
q15	518	494	480	480
q16	449	525	450	450
q17	1163	1512	1411	1411
q18	7741	7479	7415	7415
q19	827	861	1030	861
q20	1973	2115	1921	1921
q21	4892	4240	4302	4240
q22	1076	1043	972	972
Total cold run time: 52323 ms
Total hot run time: 49927 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 186650 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ce69eb4283ab724b1e5b726c5a5c5e395b364f3, data reload: false

query1	976	418	397	397
query2	6527	1867	1831	1831
query3	6753	217	219	217
query4	27017	23697	23300	23300
query5	4652	640	455	455
query6	302	201	193	193
query7	4631	505	289	289
query8	268	246	226	226
query9	8655	2640	2645	2640
query10	461	342	274	274
query11	15313	15074	14992	14992
query12	175	126	108	108
query13	1665	547	413	413
query14	9437	5938	5972	5938
query15	211	207	180	180
query16	7680	648	427	427
query17	1214	719	579	579
query18	2053	440	340	340
query19	194	186	160	160
query20	122	115	121	115
query21	214	129	111	111
query22	4067	4185	4068	4068
query23	34258	33114	33165	33114
query24	8448	2365	2413	2365
query25	541	475	403	403
query26	1226	279	149	149
query27	2696	505	341	341
query28	4259	2154	2128	2128
query29	737	550	435	435
query30	284	224	193	193
query31	965	856	758	758
query32	77	69	70	69
query33	557	387	311	311
query34	789	849	549	549
query35	812	829	741	741
query36	924	973	867	867
query37	107	98	83	83
query38	4212	4237	4141	4141
query39	1485	1432	1404	1404
query40	213	122	105	105
query41	65	61	60	60
query42	134	111	115	111
query43	510	514	481	481
query44	1317	823	818	818
query45	187	175	170	170
query46	847	1030	627	627
query47	1758	1776	1736	1736
query48	413	415	322	322
query49	751	501	389	389
query50	636	692	401	401
query51	4135	4121	4116	4116
query52	119	117	99	99
query53	235	256	185	185
query54	575	570	503	503
query55	86	84	84	84
query56	325	298	286	286
query57	1205	1204	1116	1116
query58	259	265	252	252
query59	2589	2663	2703	2663
query60	338	324	306	306
query61	145	125	125	125
query62	803	722	679	679
query63	227	193	190	190
query64	4287	1055	669	669
query65	4267	4214	4179	4179
query66	1055	414	341	341
query67	15819	15924	15286	15286
query68	7841	886	531	531
query69	456	316	271	271
query70	1199	1175	1113	1113
query71	432	327	320	320
query72	5527	4838	4897	4838
query73	700	671	356	356
query74	9002	8966	8982	8966
query75	3430	3193	2715	2715
query76	3328	1195	746	746
query77	674	386	296	296
query78	9938	10047	9386	9386
query79	2255	835	582	582
query80	571	533	454	454
query81	498	264	222	222
query82	197	134	102	102
query83	261	264	248	248
query84	264	115	95	95
query85	856	430	420	420
query86	375	317	405	317
query87	4485	4517	4501	4501
query88	3748	2261	2254	2254
query89	389	313	289	289
query90	1995	208	200	200
query91	141	148	112	112
query92	71	66	58	58
query93	1915	951	585	585
query94	669	419	316	316
query95	374	291	286	286
query96	493	576	277	277
query97	2648	2738	2691	2691
query98	236	219	203	203
query99	1317	1411	1297	1297
Total cold run time: 274549 ms
Total hot run time: 186650 ms

@airborne12
Copy link
Member Author

airborne12 commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 33777 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a778739684405b1675103160e6d5623cd4e72a7, data reload: false

------ Round 1 ----------------------------------
q1	17597	5155	5151	5151
q2	1939	307	192	192
q3	10263	1295	716	716
q4	10227	999	527	527
q5	7542	2351	2299	2299
q6	177	165	129	129
q7	877	742	606	606
q8	9315	1248	1118	1118
q9	6757	5061	5080	5061
q10	6901	2388	1977	1977
q11	480	294	267	267
q12	338	357	212	212
q13	17789	3629	3065	3065
q14	236	237	207	207
q15	565	486	479	479
q16	425	440	373	373
q17	584	854	348	348
q18	7914	7209	7029	7029
q19	1709	930	545	545
q20	326	325	215	215
q21	3703	3134	2321	2321
q22	1002	1005	940	940
Total cold run time: 106666 ms
Total hot run time: 33777 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5123	5058	5005	5005
q2	243	312	226	226
q3	2145	2607	2281	2281
q4	1338	1757	1338	1338
q5	4205	4252	4375	4252
q6	213	167	122	122
q7	2034	1912	1765	1765
q8	2540	2509	2523	2509
q9	7098	7129	7129	7129
q10	3102	3253	2796	2796
q11	589	516	504	504
q12	714	791	631	631
q13	3433	3931	3248	3248
q14	299	302	282	282
q15	542	501	487	487
q16	457	478	425	425
q17	1126	1551	1340	1340
q18	7265	7335	6926	6926
q19	748	754	853	754
q20	1887	1971	1794	1794
q21	4792	4212	4220	4212
q22	1018	1020	949	949
Total cold run time: 50911 ms
Total hot run time: 48975 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 185513 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a778739684405b1675103160e6d5623cd4e72a7, data reload: false

query1	1009	415	390	390
query2	6555	1841	1821	1821
query3	6742	214	212	212
query4	26452	23892	23119	23119
query5	4692	641	474	474
query6	323	225	201	201
query7	4627	502	292	292
query8	272	237	215	215
query9	8643	2626	2594	2594
query10	489	365	272	272
query11	15452	15143	15377	15143
query12	172	121	110	110
query13	1666	530	418	418
query14	9385	5944	5934	5934
query15	213	197	190	190
query16	7670	640	458	458
query17	1191	716	573	573
query18	2027	406	328	328
query19	204	199	162	162
query20	122	118	123	118
query21	214	131	110	110
query22	4168	4069	3913	3913
query23	33835	32902	33142	32902
query24	8447	2379	2375	2375
query25	530	469	393	393
query26	1230	264	147	147
query27	2728	501	350	350
query28	4295	2109	2082	2082
query29	745	561	423	423
query30	288	211	198	198
query31	944	842	778	778
query32	75	65	65	65
query33	556	361	311	311
query34	792	835	553	553
query35	800	825	742	742
query36	945	993	885	885
query37	110	99	81	81
query38	4154	4159	4108	4108
query39	1459	1431	1408	1408
query40	223	123	108	108
query41	62	60	68	60
query42	126	109	108	108
query43	496	509	473	473
query44	1281	804	792	792
query45	185	172	171	171
query46	844	1028	640	640
query47	1737	1780	1715	1715
query48	402	425	308	308
query49	761	481	394	394
query50	646	691	406	406
query51	4106	4190	4110	4110
query52	109	109	102	102
query53	221	255	189	189
query54	569	572	503	503
query55	87	87	83	83
query56	296	290	296	290
query57	1199	1197	1132	1132
query58	282	269	262	262
query59	2618	2704	2595	2595
query60	327	322	311	311
query61	133	127	127	127
query62	825	745	695	695
query63	230	192	190	190
query64	4287	1011	668	668
query65	4233	4160	4154	4154
query66	1076	428	313	313
query67	15940	15579	15384	15384
query68	8021	895	530	530
query69	473	301	263	263
query70	1201	1185	1062	1062
query71	471	313	301	301
query72	5360	4665	4664	4664
query73	707	583	351	351
query74	8862	9232	8962	8962
query75	3814	3221	2735	2735
query76	3644	1188	750	750
query77	799	376	297	297
query78	10031	10939	9343	9343
query79	1983	839	586	586
query80	615	502	449	449
query81	485	327	223	223
query82	446	132	98	98
query83	261	261	232	232
query84	241	115	85	85
query85	774	356	324	324
query86	390	331	296	296
query87	4547	4438	4311	4311
query88	3342	2256	2264	2256
query89	381	311	289	289
query90	1917	207	211	207
query91	145	143	116	116
query92	80	60	59	59
query93	1279	933	583	583
query94	675	412	297	297
query95	377	295	286	286
query96	489	557	279	279
query97	2757	2739	2665	2665
query98	235	216	208	208
query99	1696	1390	1275	1275
Total cold run time: 274020 ms
Total hot run time: 185513 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 29.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a778739684405b1675103160e6d5623cd4e72a7, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.25	0.06	0.06
query4	1.61	0.11	0.10
query5	0.45	0.41	0.42
query6	1.17	0.66	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.52	0.53
query10	0.58	0.57	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	0.81	0.81	0.85
query15	0.89	0.88	0.90
query16	0.38	0.40	0.40
query17	1.06	1.06	1.08
query18	0.23	0.22	0.21
query19	1.99	1.92	1.96
query20	0.02	0.03	0.01
query21	15.39	0.92	0.55
query22	0.76	1.24	0.76
query23	14.80	1.36	0.58
query24	6.66	2.13	0.75
query25	0.48	0.22	0.08
query26	0.53	0.16	0.13
query27	0.06	0.05	0.06
query28	10.28	0.92	0.45
query29	12.53	4.01	3.32
query30	0.26	0.09	0.06
query31	2.83	0.61	0.39
query32	3.28	0.56	0.47
query33	3.18	3.09	3.16
query34	16.14	5.42	4.72
query35	4.85	4.81	4.81
query36	0.70	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.15	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.35 s
Total hot run time: 29.57 s

@airborne12
Copy link
Member Author

airborne12 commented Jun 26, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 27, 2025

FE UT Coverage Report

Increment line coverage 55.88% (19/34) 🎉
Increment coverage report
Complete coverage report

eldenmoon
eldenmoon approved these changes Jun 28, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jun 30, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 5e1f24f into apache:master Jun 30, 2025
26 of 29 checks passed
@airborne12 airborne12 deleted the add-cloud-index branch June 30, 2025 03:04
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
@airborne12
[feature](index change)Support light index change for inverted index …
bf30768 
…without parser (apache#52251)

Problem Summary:
This PR adds support for lightweight index add for inverted indexes
without parser in cloud mode and extends the existing “light” mode for
NGRAM Bloom Filter indexes. And also introduces a new session variable
(enable_add_index_for_new_data) to control the above logic.
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
@airborne12
[feature](index change)Support light index change for inverted index …
37c3b1b 
…without parser (apache#52251)

Problem Summary:
This PR adds support for lightweight index add for inverted indexes
without parser in cloud mode and extends the existing “light” mode for
NGRAM Bloom Filter indexes. And also introduces a new session variable
(enable_add_index_for_new_data) to control the above logic.
koarz pushed a commit to koarz/doris that referenced this pull request Jul 4, 2025
@airborne12
[feature](index change)Support light index change for inverted index …
2f2515a 
…without parser (apache#52251)

Problem Summary:
This PR adds support for lightweight index add for inverted indexes
without parser in cloud mode and extends the existing “light” mode for
NGRAM Bloom Filter indexes. And also introduces a new session variable
(enable_add_index_for_new_data) to control the above logic.
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jul 7, 2025
@airborne12
[feature](index change)Support light index change for inverted index …
32446a3 
…without parser (apache#52251)

Problem Summary:
This PR adds support for lightweight index add for inverted indexes
without parser in cloud mode and extends the existing “light” mode for
NGRAM Bloom Filter indexes. And also introduces a new session variable
(enable_add_index_for_new_data) to control the above logic.
@airborne12 airborne12 mentioned this pull request Jul 8, 2025
Merged
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jul 9, 2025
@airborne12
[feature](index change)Support light index change for inverted index …
5c59857 
…without parser (apache#52251)

Problem Summary:
This PR adds support for lightweight index add for inverted indexes
without parser in cloud mode and extends the existing “light” mode for
NGRAM Bloom Filter indexes. And also introduces a new session variable
(enable_add_index_for_new_data) to control the above logic.
morrySnow pushed a commit that referenced this pull request Jul 9, 2025
@airborne12 @qidaye
branch-3.1: [feature](index)Support light index add for inverted inde…
b779c4a 
…x without parser and ngram bf index #48461 #52251 (#52894)

cherry pick from #48461 and #52251

---------

Co-authored-by: qiye <jianliang5669@gmail.com>
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jul 9, 2025
@airborne12 @qidaye
branch-3.1: [feature](index)Support light index add for inverted inde…
f513c2f 
…x without parser and ngram bf index apache#48461 apache#52251 (apache#52894)

cherry pick from apache#48461 and apache#52251

---------

Co-authored-by: qiye <jianliang5669@gmail.com>
@dataroaring dataroaring added dev/3.0.7-merged usercase Important user case type label and removed dev/3.0.x labels Jul 15, 2025
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@eldenmoon eldenmoon eldenmoon approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. cloud dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@airborne12 @hello-stephen @doris-robot @eldenmoon @dataroaring @gavinchou @morrySnow